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Abstract 

Background: Although the outcomes of health promotion and prevention programmes may depend on the level 
of intervention, studies and trials often fail to take it into account. The objective of this work was to develop a 
framework within which to consider the implementation of interventions, and to propose a tool with which to 
measure the quantity and the quality of activities, whether planned or not, relevant to the intervention under 
investigation. The framework and the tool were applied to data from the diet and physical activity promotion 
PRALIMAP trial. 

Methods: A framework allowing for calculation of an intervention dose in any health promotion programme was 
developed. A literature reviews revealed several relevant concepts that were considered in greater detail by a 
multidisciplinary working group. A method was devised with which to calculate the dose of intervention planned 
and that is actually received (programme-driven activities dose), as well as the amount of non-planned intervention 
(non-programme-driven activities dose). 

Results: Indicators cover the roles of all those involved (supervisors, anchor personnel as receivers and providers, 
targets), in each intervention-related groups (IRG: basic setting in which a given intervention is planned by the 
programme and may differ in implementation level) and for every intervention period. All indicators are described 
according to two domains (delivery, participation) in two declensions (quantity and quality). Application to 
PRALIMAP data revealed important inter- and intra-IRG variability in intervention dose. 

Conclusions: A literature analysis shows that the terminology in this area is not yet consolidated and that research 
is ongoing. The present work provides a methodological framework by specifying concepts, by defining new 
constructs and by developing multiple information synthesis methods which must be introduced from the 
programme's conception. Application to PRALIMAP underlined the feasibility of measuring the implementation 
level. The framework and the tool can be used in any complex programme evaluation. The intervention doses 
obtained could be particularly useful in comparative trials. 

Trial registration: PRALIMAP is registered at ClinicalTrials.gov under NCT00814554 
Keywords: Health promotion. Programme, Implementation, Dose, Process, Evaluation 
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Background 

As emphasised by Dusenbury et al [1] in their review of 
the implementation of drug abuse prevention in school 
settings, important variations in the implementation of 
interventions may arise in health promotion programmes. 
Other authors before and after him, more particularly 
Dane and Schneider [2] and Durlak et al [3] emphasised 
variation factors, such as those regarding adherence. 
Nevertheless, the outcomes of programmes are often 
analysed without taking variability of implementation 
into account [4]. This can lead to the conclusion that a 
programme is ineffective when it has not actually been 
implemented as expected ("type III error " according to 
Basch et al. [5]). In health promotion programmes, par- 
ticularly those conducted within the framework of con- 
trolled trials, it is therefore necessary to take into account 
the level to which interventions are implemented when 
interpreting outcomes. This can be viewed as the dose of 
intervention received by the target group. The interven- 
tion dose must take into account not only the activities 
performed according to the programme's frame of refer- 
ence, but also those that were conducted but not planned 
[6]. This is particularly relevant in health promotion pro- 
grammes concerning topics for which media coverage 
may lead to initiatives that are locally driven and inde- 
pendent from the planned programme. For example, nu- 
trition has been the subject of a national programme in 
France since 2001 [7]. 

The objective of the present work was to build a 
framework and to propose a tool with which to measure 
the quantity and the quality of health promotion activ- 
ities implemented, whether planned or not, related to 
the themes of the intervention under investigation. The 
framework and the tool used to assess the intervention 
dose, were applied to data from the PRALIMAP trial 
(PRomotion de I'ALIMentation et de I'Activite Physique, 
Additional file 1: Box) [8]. 

Methods 

Development of the framework 

A working group was set up. The group (comprising the 
authors of the present paper) included specialists in pre- 
vention, health promotion and health evaluation. 

A literature review also revealed several relevant 
concepts that were considered in greater detail by the 
working group. Various methods have been proposed 
with which to evaluate the process of health promotion 
programmes [1,9-17]. 

Development of the tool 

Such programmes are generally implemented in settings 
constituting homogeneous intervention groups, during 
defined period(s) of intervention. We designate them 
"intervention-related groups" (IRG). An IRG is a basic 



setting (class, school, hospital, district . . .) in which a given 
intervention (education, screening . . .) is planned as part 
of the programme and in which programme actors may 
have particular practices likely to introduce variations in 
the implementation of the activities planned within the 
programme's frame of reference and/or in the perform- 
ance of unplanned activities (beneficial or harmful in ways 
relevant to the programme). For every IRG and every 
period of intervention, the process evaluation concerned 
two major domains: the delivery of the intervention and 
the participation of those involved, each of which was 
defined in terms of quantity and quality. Four key ques- 
tions are to be answered: how much did providers do? Did 
providers do well? Did targets participate? And did targets 
participate well? 

Three categories of programme actors able to influ- 
ence implementation of health promotion programmes 
were identified [2]: supervisors, personnel anchors and 
targets. Supervisors provide personnel anchors with 
what they need to carry out the intervention, and over- 
see its implementation. Anchors have two roles: as recei- 
vers of training in the intervention by supervisors, and 
as providers of the intervention to targets. 

Crossing of both domains (delivery, participation) with 
both declensions (quantity, quality) gives four levels to 
be estimated for each type of programme actors (super- 
visor, personnel anchor (receiver and provider), target), 
i.e. 16 evaluation objects (Figure 1). In practice, only 
12 of the 16 are eligible for the process evaluation be- 
cause targets do not perform interventions and super- 
visors do not work in the field. So, indicators are 
established for every IRG in every period and bracketed in 
indicator report sheets. 

Testing the tool 

The framework and the tool used to assess the interven- 
tion dose, were applied to the PRALIMAP trial data 
Table 1. 

Results and discussion 

Results 

The framework and tool utilisation includes the follow- 
ing stages: identification of IRG, identification of inter- 
vention periods, identification and categorisation of 
programme actors, construction of indicators, data col- 
lection, data analysis and valuation of indicators, scoring, 
intervention dose calculation, and finally interpretation 
of implementation. 

Intervention Related Groups (IRG) identification 

The Intervention Related Groups (IRG) must be precisely 
identified by the investigator from the programme's incep- 
tion. They represent the possible combinations of settings 
(for example schools, hospitals, cities, districts) and 
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Figure 1 Hierarchical organisation of the 16 key objects of evaluation contributing to intervention dose calculation. 



Targets 



interventions (for example education, care, prevention) as 
defined in the programme. A setting intended to benefit 
from a particular intervention is referred to as IRG -Active 
(IRG-A), otherwise it is described as an IRG-Control 
(IRG-C) of the intervention concerned (Table 2). 

PRALIMAP 24 high schools (settings) were selected 
and three strategies (interventions) were evaluated, to 
give 72 IRG, among which 36 were IRG-A and 36 
IRG-C. 

Intervention periods identification 

When a programme is implemented, it is important to 
divide it up (particularly if it is long) into manageable 
periods in order to reduce the effects of phenomena 
affecting those involved, such as tiredness, variations in 
the learning process, and changes in personnel. 

PRALIMAP Each adolescent benefited of interventions 
over two consecutive school years (grades 10 and 11) 
corresponding to two periods. 



Identification and categorisation of the programme actors 

Depending on the programmes, three categories of rele- 
vant people (supervisors, anchor personnel, targets) may 
or may not be present. The programme investigators 
comprehensively oversee the implementation but are not 
IRG-A supervisors and must not be so defined. 

Anchor personnel receive training/information from 
supervisors, and then implement the intervention with 
the targets. They are often numerous, IRG-specific and 
occupy various posts and hierarchical positions. Infor- 
mation about events at anchor level is particularly im- 
portant because that is where potential deviations from 
a programme's frame of reference originate: deviations 
such as not performing or only partially performing 
planned activities, and introducing unplanned activities. 

Targets benefit from intervention and are the subjects 
of outcome measures. 

PRALIMAP The supervisors were the PRALIMAP 
monitors, the anchors were the high school professionals 
(administration staff, teachers, catering professionals, 
school nurses . . .) and the targets were the high school 
students. 
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Table 1 Application and adaptation to the PRALIMAP trial of the intervention dose determination framework 



Intervention Related Group (IRG) 
identification 



24 high schools * 3 strategies = 72 IRG: 36 IRG-A, 36 IRG-C 



Intervention periods identification 

Identification and categorisation of the 
programme actors 



Indicator development 



Data collection 



Data analysis and evaluation of 
indicators 



2 intervention periods = intervention implemented during tine grade 10 and 11 scliool years 
Supervisors: PRALIMAP monitors 

Anchor personnel: school professionals (administration staff, teachers, catering professionals, 
school nurses, . . .) 

Targets: high school students 

Non-programme-driven activities indicators: 

* Developed for the 72 IRG 

* Concerned respectively the educational nutritional, screening and environmental activities 
performed independently of the PRALIMAP trial 

Programme-driven activities indicators: 

* Developed for the 36 IRG-A 

* Concerned the PRALIMAP activities planned by the frame of reference: 

- 12 IRG-Education: indicators investigated the delivery of lectures and collective works on nutrition 
and the participation in PRALIMAP meetings 

-12 IRG-Screening, indicators investigated the delivery of weight and height data and of the proposition 
to participate to adapted overweight care management and the participation of students in group 
educational sessions 

- 12 IRG-Environmental, indicators investigated the delivery of high school environment improvements 
(adapted food and physical activity availability) and participation in PRALIMAP parties 

Data collected before the programme implementation: 

* High schools nutritional environment (ex: water drinking fountain, proposed physical activities . . .) : 
nutritional surveys participated in by school staff 

* Nutritional behaviours : adolescent self-administered questionnaires and anthropometric measures 
Data collected during implementation: 

* Activities delivery data: activity reports, pupil satisfaction surveys (care management, PRALIMAP 
meeting. . .) 

* Appreciation of PRALIMAP trial : self administered questionnaire 

* Evolution of the offer of school catering and physical activity free equipment and the nutritional 
environment close by the high school: nutritional surveys participated in by school staff 

Data collectedat the end of the programme: 

* Activities delivery, school staff and teenagers' participation and favouring and limiting factors : 

- focus group of staff responsible for interventional strategies (high school professionals, head teachers) 

- individual semi-structured interview of the PRALIMAP monitors 

- focus group of health professionals intervening with overweight and obese adolescents in high school 
screening 

- nutritional survey of high school professionals and students 
Indicator report sheets are elaborated for every IRG including: 

* Quantitative indicators expressed in the form of mean or percentage (eg : pupils' activity 
participation rate) 

* Qualitative (literal) indicators (eg : ranges of food proposed in the lunches, delivery or not of activity) 

The number of indicator report sheets varied from 3 to 6 according to the high school 
assigned strategies (Table 3) : 

*IRG-Education : 1 indicator report sheet of non-programme-driven activities + 1 indicator report sheet 
of programme-driven activities 

*IRG-Education control : 1 indicator report sheet of non-programme-driven activities 

*IRG- Screening : 1 indicator report sheet of non-programme-driven activities + 1 indicator report sheet 
of programme-driven activities 

*IRG-Screening control : 1 indicator report sheet of non-programme-driven activities 
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Table 1 Application and adaptation to the PRALIMAP trial of the intervention dose determination framework 

(Continued) 

*IRG-Environnnent : 1 indicator report slieet of non-programme-driven activities + 1 indicator report slieet 
of programme-driven activities 

*IRG-Environment control : 1 indicator report slieet of non-programme-driven activities 

Score assignment Number of experts:! 8 (3 groups of 6) 

Type and specialty of experts: researcliers, field professionals or decision-makers, specialists in diet, 
physical activity andXor evaluation, knowing or not the PRALIMAP trial, practicing or not in Lorraine 
Region 

IRG assigned between the experts: the IRG were fairly and anonymously distributed among the 
experts 

Individual scoring aid: IT (Excel®) 

Scoring : ranging from 0 to 20 for every period, domain and characteristic in each IRGThreshold 

defined for the standard deviation and/or the range: if a standard deviation was higher than 2.5 or a 
range higher than 6 was observed, the experts debated and proposed a new notation; discrepant scores 
were then preserved. 

Taking into account between-group variability: A fictitious high school was created and scored by 
the 3 groups 

Intervention dose calculation Application of intervention dose formula to assigned scores: Dose = DQt x (mean (DQI, PQt 

PQI)/20) 

A group effect has been evidenced thanks to the fictitious high school and required score adjustment 
varying from 0.8 to 2.8 points. 

Eventually 216 doses (108 per period) were calculated (Table 3). 



Indicators development 

Two types of indicator are required: specific indicators 
related to programme-driven activities, and general indica- 
tors related to non-programme-driven activities. The latter 
may lead to over- or under-estimation of programme- 
driven activities due to synergy or antagonism, respectively. 

Programme-driven activities indicators were established 
for IRG -A, for every period and each of the 16 evaluation 
objects (Figure 1). Non-programme-driven activities indi- 
cators were developed for every IRG (including IRG-C, if 
any), every period, and every evaluation object. 

PRALIMAP Non-programme-driven activities indica- 
tors were developed for the 72 IRG and concerned the 

Table 2 An example of intervention related group (IRG) 
identification 

Setting 1 Setting 2 Setting 3 IRG 

Intervention 1 Yes Yes No 3 IRG : 

2 IRG- Active 

1 IRG-Control 

Intervention 2 No No Yes 3 IRG : 

1 IRG- Active 

2 IRGs-Control 

IRG 2 IRG : 2 IRG: 2 IRG : 6 IRG : 

1 IRG- Active 1 IRG- Active 1 IRG-Active 3 IRG- Active 
1 IRG-Control 1 IRG-Control 1 IRG-Control 3 IRG-Control 



educational nutritional, screening and environmental ac- 
tivities performed independently of the PRALIMAP 
trial. Two examples of this type of activities can be 
given : eco-citizenship actions around nutrition took 
place in some of schools in the frame of the Agenda 
21' plan ; actions (Sport, Wellness, first aid, breakfast, 
fruit. . .) has been implemented by some school staffs 
during local initiatives such as a 'health week'. 

Programme-driven activities indicators were estab- 
lished for the 36 IRG -A and concerned the planned 
PRALIMAP activities. Twelve IRG education indicators 
investigated the delivery of lectures and collective work 
on nutrition and participation in PRALIMAP parties. 
Twelve IRG screening indicators investigated the collec- 
tion of weight and height data and information about 
intention to participate in adapted care management and 
the participation of students in group educational ses- 
sions. Twelve IRG environment indicators investigated 
improvements at high schools (changes in diet and phys- 
ical activity available) and participation in PRALIMAP 
parties. 

Data collection 

Data collection relied on regular activity reports and on 
quantitative and qualitative investigations. 

Activity reports permit monitoring of the quantity of 
intervention delivered and of participation in activities. 
They must be regularly completed by supervisors and 
providers. 
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The quantitative investigation of large target popula- 
tions generally involves self-administered questionnaires, 
ideally completed at the same time as outcome measure- 
ment. It retrospectively assesses what has been done be- 
tween two outcomes measurement points. 

Qualitative investigation allows for measurement of 
delivery and participation and elucidates the interpreta- 
tions and points of view of those involved. Collection 
methods are generally observation, collective interview 
(such as focus groups) and individual interview [3,18,19] 

Both types of investigation complement one another and 
involve collection of information from the various people 
involved for every IRG and every period of intervention. 

Data can be collected at various points: 

- before programme implementation to provide 
information about the initial context 

- during implementation, at the end of every period, 
to compare (in a concomitant or retrospective way) 
the performed activities to planned ones and to 
identify performed but not planned activities 

- at the end of the programme to assess general 
response and satisfaction. 

The programme actors involved are the objects and the 
sources of information. For example, targets may report 
on their own participation and that of anchors. 

PRALIMAP Before the programme implementation, nu- 
tritional environmental data were collected at high schools 
via surveys of the staff. During implementation, delivery 
data were included in activity reports. Student satisfaction 
with the programme was measured using a self- 
administered questionnaire completed at the same time as 
outcome measurement and surveys of satisfaction with 
specific activities (care management, PRALIMAP party). 
Information about changes in school catering and physical 
activity supply, availability of free equipment, and the nu- 
tritional environment in the neighbourhood of the high 
school was assessed with a survey among the high school 
professionals. At the end of the programme, data on activ- 
ity delivery, and on participation by school staff and stu- 
dents were collected by focus groups of staff responsible 
for interventional strategies (high school professionals, 
head teachers), and by individual semi-structured interview 
of PRALIMAP monitors. 

Data analysis and valuation indicators 

Data analysis allowed for valuation of the indicators devel- 
oped. To facilitate the later expertise work, the valued 
indicators are bracketed within indicator report sheets. 
For every IRG, one or two indicator report sheets were 
elaborated, one covering non-programme- driven activities 
indicators and the other the programme-driven activities 



indicators (if IRG-A). On every indicator report sheet 
(Figure 2), indicators were presented by domain, de- 
clension, and programmes actors concerned as object 
and source of information, for each period of intervention. 

PRALIMAP Indicator report sheets were developed for 
every IRG and included quantitative indicators expressed in 
mean or percentage (eg: pupils' activity participation rate), 
qualitative (literal) indicators (eg: ranges of food proposed for 
lunches, delivery or not of activity). The number of indicator 
report sheets varied from three to six according to the high 
school assigned strategies (Table 3) totalling 72 indicator re- 
port sheets of non-programme-driven activities indicators 
and 36 of programme-driven activities indicators (IRG-A). 

Assignment of scores 

We used the nominal group technique [20] to reach con- 
sensual scores. A score covers an IRG set of programme- 
driven or non-programme-driven activities indicators; it is 
assigned for every domain / declension and every period 
(Figure 2). It is impossible to establish from indicators (in 
particular those stemming from a qualitative investigation) 
an automatic scoring system. Collective expert techniques 
are the best methods in that context [21]. 

The collective expertise method is multidisciplinary, in- 
cluding decision-makers, professionals, researchers, and 
specialists in the topic of interest and\or the evaluation. 
The experts do not all have to be actively involved in the 
programme being assessed. Depending on the number of 
IRG concerned and available resources, one or several 
groups of at least six experts are constituted so as to 
obtain a variety of opinions. The notation sessions are 
managed by an independent moderator and take place 
in the following way: 

- anonymous presentation of the IRG characteristics 
to provide the experts with an overview of the 
environment in which the programme took place, 

- explication of the indicators and indicator report sheets, 

- determination of a theoretical range of scores, 

- IRG-blind scoring by the experts on an individual 
marking aid (IT or paper). 

The mean, standard deviation and the range of scores 
assigned by the experts are calculated for every domain/ 
declension and every period. If the standard deviation 
and/or the range exceed a previously agreed threshold, 
experts debate (under the moderator) in order to explain 
the deviations, and to look for a possible consensus. The 
debate leads to a second score. Mean scores are then 
preserved even in the absence of consensus [22]. 

When several groups of experts are constituted and in 
order to take into account the between-group variability, a 
fictitious IRG can be proposed to allow for a calibration. 
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Indicators 


First Period 




Observation 




The delivery of the intervention — ^ 

Quantity — __^^Declension 


domain 


Supervisors' information v^IITli; 


As a whole have the planned activities been 
implemented ? 

Have the planned activity 1 

been implemented ? 

Have the planned activity 2 

been implemented ? 




Source of 

information J 


Answers: may be expressed on a 
binary scale (yes/no) or a Likert 
scale (totally, partially, almost not) 
or in percentage 


Anctiormen' information 


As a whole have the planned activities been 
implemented ? 






Answers : may be expressed on a 
binary scale (yes/no) or a Likert 
scale (totally, partially, almost not) 
or in percentage 


Target persons' information 


As a whole have the planned activities been 
implemented ? 






Answers : may be expressed on a 
binary scale (yes/no) or a Likert 
scale (totally, partially, almost not) 
or in percentage 


Delivery quantity mark ( /maximal assignable 
mark) 








The delivery of the intervention 
Qualtity 


Supervisors' information 


As a whole have the objectives been achieved ? 

Have the activity 1 

been produced any 

perceptible change ? 

Have the activity 2 

been produced any 

perceptible change ? 






Answers: may be expressed on a 
Likert scale (totally, partially, almost 
not) or in percentage 


jntified activites 








Anctiormen' information 


As a whole have the objectives been achieved ? 






Answers: may be expressed on a 
Likert scale (totally, partially, almost 
not) or in percentage 


Target persons' information 


As a whole have the activites been useful?** 






Answers: may be expressed on a 
Likert scale (totally, partially, almost 
not) or in percentage 


Delivery quality mark ( /maximal assignable 
mark) 








Participation in the activities 
Quantity 




Supervisors' information 


As a whole how many anchormen received the 
formation ? 

As a whole how many anchormen implemented 
the activity near the target persons? 

As a whole how many target person received 
the intervention ? 






percentage 
percentage 
percentage 


Anctiormen' information 


As a whole how many target persons received 
the intervention ? 






percentage 


Target persons' information 


As a whole how many target persons declared 
having participated? 






percentage 


Participation quantity mark ( /maximal 
assignable mark) 








Participation in the activities 
Quality 




Supervisors' information 


As a whole have the target persons participated 
well ? 






Answers: may be expressed on a 
Likert scale (totally, partially, almost 
not) or in percentage 


Anchormen' information 


As a whole have the target persons participated 
well ? 






Answers: may be expressed on a 
Likert scale (totally, partially, almost 
not) or in percentage 


Target persons' information 


As a whole how much target persons were 
satisfied with the intervention?**: 






Answers: may be expressed on a 
Likert scale (totally, partially, almost 
not) or in percentage 


Participation quality mark ( /maximal 
assignable mark) 









Figure 2 Template of indicators report sheet. 
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Table 3 Number and type of IRG and number of indicator report sheets and scores according to the high school and 
its assigned PRALIMAP strategies 

N° school Strategy Indicator report sheets 

Education Screening Environment Non-programme-driven Programme-driven Score total / school 











activities 


activities 




/ 


IRG-C 


IRG-C 


IRG-C 


3 


0 


3 


2 


IRG-A 


IRG-A 


IRG-C 


3 


2 


5 


3 


IRG-A 


IRG-A 


IRG-A 


3 


3 


6 


4 


IRG-A 


IRG-C 


IRG-A 


3 


2 


5 


5 


IRG-A 


IRG-C 


IRG-C 


3 


1 


4 


6 


IRG-A 


IRG-A 


IRG-A 


3 


3 


6 


7 


IRG-A 


IRG-A 


IRG-C 


3 


2 


5 


8 


IRG-C 


IRG-A 


IRG-C 


3 


1 


4 


9 


IRG-A 


IRG-A 


IRG-A 


3 


3 


6 


10 


IRG-C 


IRG-A 


IRG-A 


3 


2 


5 


11 


IRG-A 


IRG-C 


IRG-C 


3 


1 


4 


12 


IRG-C 


IRG-A 


IRG-C 


3 


1 


4 


13 


IRG-C 


IRG-C 


IRG-C 


3 


0 


3 


14 


IRG-A 


IRG-C 


IRG-A 


3 


2 


5 


15 


IRG-C 


IRG-A 


IRG-A 


3 


2 


5 


16 


IRG-C 


IRG-C 


IRG-A 


3 


1 


4 


17 


IRG-A 


IRG-A 


IRG-C 


3 


2 


5 


18 


IRG-A 


IRG-C 


IRG-A 


3 


2 


5 


19 


IRG-C 


IRG-C 


IRG-A 


3 


1 


4 


20 


IRG-C 


IRG-C 


IRG-C 


3 


0 


3 


21 


IRG-C 


IRG-C 


IRG-A 


3 


1 


4 


22 


IRG-C 


IRG-A 


IRG-A 


3 


2 


5 


23 


IRG-A 


IRG-C 


IRG-C 


3 


1 


4 


24 


IRG-C 


IRG-A 


IRG-C 


3 


1 


4 




12 IRG Education 


12 IRG Screening 


72 IRG Environment 


72 


36 


108 



1 2 IRG -Control - 1 2 IRG -Control- 1 2 IRG -Control- 
Education Screening Environment 



Eventually, four IRG^period non-programme-driven 
activities scores and four IRG-A*period programme- 
driven activities scores are obtained for every period. 

A wrap up debate needs to be performed with all the 
experts, in particular if several groups have been consti- 
tuted. It allows for discussion of the relevance of scores, 
the difficulties encountered and the between-group vari- 
ability, and preparation for the formal weighting of 
scores to be used for the dose calculations. 

PRALIMAP Three groups of six experts were consti- 
tuted, comprising: 

- researchers, field professionals or decision-makers, 

- specialists in food, physical activity and/or 
evaluation, 

- people familiar or not with the PRALIMAP trial. 



- people practising or not in the Lorraine Region. 

The experts assigned scores ranging from 0 to 20 for every 
period, domain and declension in each of the IRG, distribu- 
ted fairly and anonymously among the experts. The scores 
were entered on computers, allowing for immediate display 
of results. If a standard deviation higher than 2.5 or a range 
higher than 6 was observed, the experts debated. A fictitious 
high school was created and scored by the three groups. 

Intervention dose calculation 

The four declensions are not independent but nested: 
participation is subject to delivery, and quality is subject 
to quantity. In practice, the impact of the quantity of activ- 
ity delivery is likely to be decreased by the delivery quality 
as well as by the quantity and quality of participation. The 
delivery quantity score is therefore weighted by the mean 
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of the delivery quality scores and of the participation 
quantity and quality score, the mean being divided by the 
common maximal assignable score. 

Thus Dose = DQt x (mean (DQl, PQt, PQl)/mas): 
DQt 

delivery quantity score 

DQl 

delivery quality score 

PQt 

participation quantity score 

PQl 

participation quality score 

mas 

common maximal assignable score 

Two doses are calculated for every intervention period: 
one non-programme-driven activities dose for every IRG 
and one programme-driven activities dose for every 
IRG-A. 

PRALIMAP The formula was applied to scores assigned 
to each of the 72 IRG covering the 24 high schools. 
Overall, 216 doses (108 per period) were calculated: four 
for every IRG-A (a non-programme-driven activities 
dose and a programme-driven activities dose for each of 
the two periods) and two doses for every IRG-C (a non- 
programme-driven activities dose for each of the two 
periods). 

A group effect revealed by the fictitious high school neces- 
sitated score adjustment varying from 0.8 to 2.8 points. 

Implementation interpretation 

The unit of analysis is the setting. For each setting, clus- 
ter characteristics (e.g. geographical zone, socioeconomic 
status) and target population characteristics (e.g. sex 



ratio, mean age, total number of professionals) were col- 
lected. Doses are expressed as means, medians, and dis- 
tribution parameters. Doses calculated for an IRG are 
assigned to every target person belonging to it. The ana- 
lysis allows for dose comparisons between IRG or IRG 
clusters as defined in the outcomes analysis plan. 

PRALIMAP The twelve mean doses (four for each 
of the three PRALIMAP strategies) obtained ranged 
from 5.2 (programme-driven activities dose second year 
screening) to 9.0 (non-programme-driven activities dose 
first year education) (Table 4). 

Variability of delivery from one high school to the 
other was evidenced for all the strategies; nutritional 
educational activities were performed in all the high 
schools allocated or not to the education strategy. A few 
active high schools performed practically no activity, in 
particular for the screening strategy. 

The mean doses were low. The programme-driven ac- 
tivities mean dose of IRG-education for the first year 
was 8.2, while the mean doses of four constituent 
declensions / characteristics varied from 11.4 for the 
participation quality to 13.6 for the delivery quantity, 
with an IRG dose range from 6.5 for participation quan- 
tity to 15.8 for delivery quality (Table 5). 

The mean dose was higher in the first year than the 
second, with the exception of the programme-driven ac- 
tivities dose of the environment strategy. The median 
was lower than the mean except for environment. Con- 
trol high schools had non-null intervention doses that 
were weak for the screening strategy (2.7 year 1), 
and higher for environment (5.6) and education (6.3). 
They benefited from interventions not planned by the 
programme. High schools that benefited from a strategy 
had doses significantly higher than their controls whatever 



Table 4 Global description of the intervention doses in the 24 high schools participating in the PRALIMAP trial 









N 


mean 


standard deviation 


median 


Q1 


Q3 


min 


max 


Education 


NPDA* 


year 1 


24 


9.0 


3.5 


8.3 


6.6 


12.5 


3.4 


13.9 






year 2 


24 


7.7 


2.8 


7.8 


5.5 


10.0 


2.5 


13.2 




PDA** 


year 1 


12 


8.2 


1.9 


7.8 


7.3 


9.3 


4.7 


11.8 






year 2 


12 


6.3 


3.5 


7.5 


2.7 


8.5 


0.5 


10.9 


Screening 


NPDA 


year 1 


24 


5.2 


3.7 


5.0 


2.3 


8.4 


0.0 


11.9 






year 2 


24 


5.0 


3.4 


5.4 


1.8 


7.9 


0.0 


10.1 




PDA 


year 1 


12 


6.3 


3.5 


6.6 


3.1 


9.3 


1.2 


10.8 






year 2 


12 


3.6 


2.7 


3.0 


1.9 


4.8 


0.5 


8.8 


Environment 


NPDA 


year 1 


24 


6.4 


2.3 


7.0 


5.8 


7.8 


1.8 


9.4 






year 2 


24 


5.7 


2.0 


6.1 


4.6 


7.3 


1.2 


9.6 




PDA 


year 1 


12 


7.8 


1.8 


7.6 


6.8 


8.4 


5.2 


12.5 






year 2 


12 


8.2 


3.1 


8.6 


6.9 


9.7 


0.3 


12.1 



* NPDA : Non-programme-driven activities. 
PDA : Programme-driven activities. 
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Table 5 IRG Education - year 1 detail of the mean assigned marks 







N 


mean 


standard deviation 


median 


Q1 


Q3 


min 


max 


Delivery 


quantity 


12 


13.6 


1.5 


14.2 


12.4 


14.8 


10.5 


15.7 


Delivery 


quality 


12 


12.2 


2.6 


12.1 


10.6 


14.5 


8.2 


15.8 


Participation 


quantity 


12 


12.2 


2.4 


12.5 


11.2 


13.6 


6.5 


15.2 


Participation 


quality 


12 


11.4 


2 


11.2 


9.8 


12.8 


8.7 


14.5 



the year. The general environment dose was significantly 
higher in IRG -A education than in IRG-C-education, and 
the educational non-programme-driven activities dose in 
the second year was significantly higher in IRG -A environ- 
ment than IRG-C environment (Table 6). 

A significant negative interaction between the educa- 
tion and environment strategies emerged (Figure 3). 
When education and environment were implemented in 
combination, the doses of both were lower than expected 
in an additive model The screening strategy was imple- 
mented independently of the other strategies (absence of 
interaction). 

A multivariate analysis taking into account cluster 
characteristics (implementation waves, high school edu- 
cation type and geographical zone) and individual char- 
acteristics (gender, age, social and occupational status, 
BMI) did not modify the results. 

Discussion 

A framework and a tool allowing for calculation of an im- 
plementation dose of programme- or non-programme- 
driven activities during health promotion programmes were 
elaborated, investigated and validated in a cluster rando- 
mized trial. An approach led by the theory necessitated spe- 
cification of certain concepts (dose, delivery, participation, 
quantity, quality, programme actors, information sources), 
definition of new constructs (IRG, period, programme- 
driven activities, non-programme-driven activities) and de- 
velopment of information synthesis techniques (indicator 
report sheets by IRG, collective expertise, practical details 
of intervention dose calculation). Application in PRALI- 
MAP confirmed the feasibility of the approach, demon- 
strated important implementation variability between IRG 



and over time, and showed that intervention doses can be 
obtained and used in future 'in treatment' analysis. 

The importance of process in health programmes and 
trials has been increasingly recognised in recent decades 
and has been the subject of three important reviews 
[1-3]. In 1998, Dane and Schneider, reviewed 162 primary 
and secondary prevention studies [2]. They emphasised 
that failure to consider integrity data, particularly regard- 
ing adherence, can compromise the internal validity of 
prevention studies. In 2003, Dusenbury et al. [1], analyzed 
drug addiction prevention studies performed over a 
25-year period. They revealed that poor implementa- 
tion may reduce a programme's effectiveness and that 
strong methodologies to measure and analyze imple- 
mentation should be developed. In 2008, Durlak et al. 
[3] reviewed more than 500 articles (the majority of 
which were already synthesised in five meta-analyses) 
and clearly showed that implementation level affects 
the outcomes of health promotion and prevention pro- 
grammes. They contributed to the description of fac- 
tors that influence implementation, and recommended 
implementation data collection, which they consider 
an essential feature of programme evaluation. 

These three reviews showed that the terminology is 
not yet consolidated, probably hindering the dissemin- 
ation of data from implementation studies. We used a 
pragmatic and general classification that covered the 
concepts used in the reviews (Table 7). They sometimes 
proposed components not of the same nature, for ex- 
ample exposure and programme differentiation. Indeed, 
programme differentiation is a peculiar characteristic 
that can influence implementation but does not depend 
on programme actors, whereas exposure represents the 



Table 6 Mean doses obtained for each of the three PRALIMAP strategies 



Dose 



EDUCATION 



SCREENING 



ENVIRONNEMENT 



Control 



Active 



Control 



Active 



Control 



Active 







NPDA 


PDA 


NPDA 


PDA 


NPDA 


PDA 


NPDA 


PDA 


NPDA 


PDA 


NPDA 


PDA 


Education 


year 1 


6,3 




^^,7 


8,2 


9,2 


8,8 


8,9 


7,6 


8,2 


7,7 


9,9 


8,7 




year 2 


6,0 




9,5 


6,3 


8,0 


6,2 


7,5 


6,5 


6,8 


5,5 


8,6 


7,2 


Screening 


year 1 


6,1 


6,4 


4,4 


6,2 


2,7 




7,7 


6,3 


5,5 


6,3 


4,9 


6,3 




year 2 


5,2 


3,7 


4,7 


3,4 


2,5 




7,4 


3,6 


5,2 


3,4 


4,8 


3,8 


Environment 


year 1 


5,2 


8,0 


7,6 


7,6 


6,4 


8,2 


6,4 


7,4 


5,6 




7,2 


7,8 




year 2 


4,9 


8,9 


6,6 


7,5 


5,7 


8,3 


5,7 


8,0 


4,9 




6,6 


8,2 



Bold face values: statistically significant difference of the received dose between the control and the strategy groups. 
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Interaction between the education and environment 
strategies Educational intervention doses - first year 
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Interaction between the education and environment 
strategies 

Educational intervention doses - second year 



9,1 



9,8 



♦ 7,4 



♦ 4,5" 



-♦- -Education- 
— ■ Education+ 



Environment - 



Environment + 



Interaction between the education and environment 
strategies Environmental intervention doses - first year 
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Environmental intervention doses - second year 
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Figure 3 PRALIMAP educational and environmental intervention dose received according to the assigned strategies. 



amount of the programme delivered. It is not always 
easy to distinguish, in papers, what is exactly meant by 
adherence, dose or quality. Our classification allows for 
a hierarchical organization of four components and thus 
for the calculation of what we call the 'intervention 
dose'. These components are obtained by simply answer- 
ing four questions: how much did providers do? Did 
providers do well? Did targets participate? And did tar- 
gets participate well? 

Like Dane and Schneider [2], we put the emphasis on 
clearly identifying, during the indicator construction 
process, the information sources and the various personnel 
involved in the programme - each of whom might be a 
source of information on the others. For example, in a 
school programme, students may assess the teachers' par- 
ticipation and vice versa. That is why we suggest precisely 
identif)Aing the people associated with each of the four 
components (Figure 1). 

As underlined by Durlak et al. [3], no study has reported 
100% implementation by providers. The implementation 
level depends on supervisors or providers, and varies from 
20 to 40% depending on the setting. A supervisor or a pro- 
vider operating in several programme settings can even 



behave differently in each. So it seems necessary to take 
into account the setting- and intervention-specific imple- 
mentation level; hence we elaborated the new concept of 
IRG. In PRALIMAP, the variety, the number of supervisors 
and providers and the potential substitution of individual, 
from one school year to the next brought to light the im- 
portance of taking into account the period and the IRG. 
For the evaluation of an effectiveness trial, this notion is 
crucial to understanding of the relation between the imple- 
mentation and the outcomes. It is just as important in 
health programmes not in the context of a trial in order to 
take account of variability and weaken the dilution effect 
induced by heterogeneity of settings. 

Most studies consider only those activities directly driven 
by the programme. Durlak suggests considering the contam- 
ination aspect (treatment contamination, usual care, alterna- 
tive services) in the level of implementation assessment, 
particularly when a control comparative group is used. We 
stress that implementation in a specific programme may be 
influenced by other concomitant programmes such as na- 
tional media campaigns, local programmes or personal 
initiatives by those involved in the programme under consid- 
eration. Therefore, we distinguished between programme- 



Table 7 Correspondence between the concepts used in this paper and three reviews 



Reviews 




Dane and Schneider [2] 


Dusenbury et al. [1] 


Durlak et al. [3] 


Legrand et al. 










Intervention dose 


Integrity or Fidelity including 5 components : 

exposure, adherence, quality of delivery, 
programme differentiation, participant 
responsiveness 


Fidelity including 5 components: 

adherence, dose, quality of delivery, 
programme differentiation, participant 
responsiveness 


Including 8 coas mponents: 

Fidelity, Dosage, Quality, 
Participant responsiveness. 
Programme differentiation. 
Contamination, 
Programme reach. 
Programme modification 


Delivery 


Quantity 


Exposure 

A pi hprppirp 
/\U 1 ICI CI l>^C 


Dose 

A pi hprpnrp 

/\U 1 ICI CI l>^C 


Dosage 

Fidelity (a k a : adherence or 
compliance or integrity, or 
faithful replication) 




Quality 


Quality of delivery. Adherence 


Quality of delivery. Adherence 


Oi 1 3 lit\/ 

v^uaiiLy 


Participation 


Quantity 


Participant responsiveness 


Participant responsiveness 


Programme reach 




Quality 


Participant responsiveness 


Participant responsiveness 


Participant responsiveness 


Participants/ sources of 
information 


Supervisors 


Supervisors, 

Developers 
Facilitators 








Anchors personnel 
(providers / receivers) 


Implementers (receivers) or providers 


Providers 


Providers 




Targets 


Participants 






IRG 




/ 


/ 


/ 


Indicators 


Non-programme- 
driven activities 


/ 


/ 


/ 




Programme-driven 


/ 


/ 


/ 



activities 
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driven and non-programme-driven activities relevant to the 
intervention under investigation. 

PRALIMAP showed not only the importance of this 
distinction, particularly when estimating the effect of the 
intervention, but also the difficulty of distinguishing 
whether an activity (for example the delivery of a nutri- 
tion course within the curriculum) is performed in the 
programme's frame of reference. So, in high schools ac- 
tive for a given strategy, the non-programme-driven ac- 
tivities scores were higher than in control high schools 
when we could have expected them to be equal or even 
lower. 

It is essential to have in mind the indicators from the 
programme inception to be sure to eventually have indi- 
cators for every domain, every declension and every per- 
son involved; the quality and the sufficiency of the data 
collection depend on that. So in PRALIMAP we were not 
able to collect data on participation in non-programme- 
driven screening activities. 

Collective expertise appeared to be the most appropri- 
ate method [20,22,23] with which to facilitate dose cal- 
culation. In PRALIMAP, the experts underlined the 
importance of the first indicator sheet, which acts as a 
scoring reference. We observed between-group variabil- 
ity in scores thanks to the fictitious high school. To min- 
imise variability, we recommend limiting the number of 
expert groups and submitting to the experts (without 
their knowing) a first indicator sheet corresponding to a 
fictitious IRG, which allows for measurement of the 
group effect and, if necessary, adjustment of scores. 

Application to PRALIMAP confirms our hypothesis of 
strong implementation variability between IRG, with de- 
viation depending on period and intervention strategy. 
Awareness of this variability is necessary in order to esti- 
mate the influence of implementation on programme 
outcomes [3]. That will be performed in PRALIMAP by 
on-treatment' analysis [24], in which the calculated dose 
of an IRG will be assigned to each student of that IRG. 
It is thus about a dose calculated collectively and not in- 
dividually. The variability of the calculated final IRG 
dose may depend on the weighting method. The method 
we used reflects at best the implementation level in the 
target population but tends to reduce the dose measure 
variability. The 'in treatment' analysis could allow for 
validation of the proposed weighting method. 

Conclusions 

The implementation of complex public health and 
health promotion programmes is measurable thanks to 
the calculation of an intervention dose. The calculation 
is based on the construction of indicators developed 
from the conception of the programme and rigorous 
data collection on the processes with programme actors 
likely to induce variations in the implementation. 



Independent collective expert input ensures the valid- 
ity of the measure obtained. 

The tool can be used in any programme evaluation. It 
could be particularly useful in comparative trials and in 
studies of the influence of implementation on programme 
outcomes. Further developments and researches are needed 
to ensures its utility and evaluate its transferability to other 
contexts. 

Additional file 



Additional file 1: Box. The PRALIMAP trial. 
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